Error Tight
Monitoring our research pipelines
We all make mistakes
Today
•
Story time"
•
10 min on with your group"
•
10 min on with the class"
•
X3
•
Story 1 - beta testing"
•
Story 2 - rewriting code
Me
Staff Scientist
•
[An analysis] was coming out#too good; ...some of
the effects were just implausibly large, and not
affected by the conditions like they should have
been. ...We discovered it was a communication error
resulting in incorrect condition labels. I hadn't
preprocessed the dataset myself; a colleague ran the
experiment at a different university and we had
several meetings about its files. ...I coded up the
conditions as described in my notes from the
meetings ... but my notes were wrong."
•
This only resulted in lost time and aggravation. But it
was really tricky, since some of the results came out
better, not worse, and the error was in my initial task/
file description (so wasn't found by me comparing
the code with my notes - those matched)
Jo Etzel, PhD
Prof of Psychiatry
Co-chair of DBBS Curriculum
•
I [a grad student] was characterizing the % of
hippocampal neurons in culture that used
glutamate vs. GABA. My PI decided to help with
increasing the N. My own data suggested that
~80% of neurons used glutamate. However,
when my PI started recording, he got nearly
reversed results from mine – nearly 80%
GABAergic. We eventually traced the
discrepancy to unconscious biases we each had
in cell selection. Each of us was conditioned by
previous studies we had performed to overlook
cells of a certain shape. Ultimately, I repeated the
experiment, forcing myself to record each and
every cell that I encountered. The final result was
approximately half and half. It was a lesson in
unconscious bias/data
Steve Mennerick, PhD
Consultant
•
Kelli joined a lab that had the following
predicament:"
•
Postdoc worked in lab, collected and
managed all the data. Data were
password protected."
•
Said postdoc gets a new job and
leaves the lab (standard practice)"
•
Password goes with the postdoc. They
can't reach the postdoc."
•
The data was locked away for ~3 years
before it was able to be accessed and
analyzed.
Kelli Huber, PhD
Prof of Psychology
Josh Jackson, PhD
Prof and Chair of
Organizational Behavior
1. I just got a paper accepted to PSPB in which we have a footnote saying that
a record-keeping error caused us not to be able to report reliability for one of
our variables. The variables in this study were painstakingly coded by an
army of undergrads who read the transcripts of people negotiating. The two
graduate students who ran the coding process both left academia and didn’t
keep their files. We had the reliability data about all the variables except one,
by digging out an early version of the manuscript write-up, but one variable
was added later and now we’ll never know. The lesson here is for multiple
collaborators to keep the data in their files, although I’ll admit that
sometimes I still neglect to do this with student coauthors and really should."
2. I had a grad school classmate who coded participant “sex” as 0/1 and later
forgot which direction was which. The lesson here is always to code sex as
“female” (or “male”) 0/1, which also makes it easier for the readers."
3. My grad school mentor withdrew an accepted paper from a top journal
because she realized that one of the variables was coded backwards.
Hillary Anger Elfenbein
Step 1: Designing/programming experiment
With Groups
•
In your group, the person who was assigned this topic please describe how
you (or your lab) goes about this. If you do not have an active research
program, defer to someone who does have an active research program."
•
Your group needs to come up with ideas on where errors could come up in
this area"
•
Someone in the group take notes
Step 1: Designing/programming experiment
With Class
•
What did you come up with? "
•
How do we avoid these?
Step 2: Collecting Data
With Groups
•
In your group, the person who was assigned this topic please describe how
you (or your lab) goes about this. If you do not have an active research
program, defer to someone who does have an active research program."
•
Your group needs to come up with ideas on where errors could come up in
this area"
•
Someone in the group take notes
Step 2: Collecting Data
With Class
•
What did you come up with? "
•
How do we avoid these?
Step 3: Analyzing Data/Reporting Findings
With Groups
•
In your group, the person who was assigned this topic please describe how
you (or your lab) goes about this. If you do not have an active research
program, defer to someone who does have an active research program."
•
Your group needs to come up with ideas on where errors could come up in
this area"
•
Someone in the group take notes
Step 3: Analyzing Data/Reporting Findings
With Class
•
What did you come up with? "
•
How do we avoid these?
